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I. Introduction 


The cost of attending college has risen sharply over the last 40 years. Although more credit and 
grant aid have been made available to students, there are still major gaps between aid and the 
cost of attendance for many students in the United States, all of whom are left to figure out 
whether they can afford the remaining costs associated with attending college. Given that 
people with similar financial means may make different decisions about what they can and 
cannot afford to pay for college, it is difficult to identify a universal definition of affordability. 
Nevertheless, examining college cost, student financial need, and progress toward degree 
completion trends in concert with one another can help illuminate patterns of when college is 
unaffordable. This paper will develop a definition of college affordability by examining the 
literature on student financial need and student aid interventions as well as student data collect- 
ed by MDRC to identify important indicators of college affordability and the lack thereof, 
which may ultimately help or impede college persistence and graduation. 

The average cost of a public four-year institution increased from $617 to $5,491 in 
nominal terms (or by 270 percent adjusted for inflation) from 1976 to 2005. 1 Furthermore, in 
the five years spanning the 2007-2008 academic year to the 2012-2013 academic year, tuition 
and fees at four-year public universities increased by an additional 27 percent. 2 On average, 17 
percent of net family income was needed to cover the cost of attending a public four-year 
institution after all financial aid received was applied in 2009. In contrast, among families in the 
lowest income quintile, over 60 percent of net family income was needed to cover the average 
cost of attending a four-year institution. 3 

Given the increasingly high relative cost of attending college for families at the low end 
of the income scale, it is not surprising that students from such backgrounds are less likely to 
stay enrolled in college than their higher-income peers. In fact, enrollment after the first year is 
about five times higher for high-income students than for low-income students. 4 Titus’ study of 
college degree completion found that 7 1 percent of students in the highest economic quartile 
completed college within six years, but only 46 percent of students in the lowest economic 
quartile completed college within the same time frame. 5 

While academic preparation cannot be discounted as a key factor in college success, the 
negative consequences of high levels of unmet financial need on student success are likely to be 


'Long and Riley (2007). 

2 College Board (2012). 

'National Center for Higher Education Management Systems (2013). 
4 Chen and Zerquera (2011). 

5 Titus (2006). 
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greater for lower-income students than for higher-income students. 6 Long and Riley (2007) 
found substantial increases in umnet need from 1995-1996 to 2003-2004 for all full-time, full- 
year undergraduates. Moreover, these increases in umnet need were highest for low-income 
students. Over this period, there was a 68 percent increase in unmet need for low-income 
students attending two-year institutions, a 59 percent increase for these students attending public 
four-year schools, and a 49 percent increase at private four-year institutions. Figure 1 displays 
the umnet financial need averages for independent and dependent full-time, full-year students at 
both public and private four-year and two-year institutions after grant aid has been applied. For 
independent and dependent students in the highest income quartile, umnet need ranged, on 
average, from $6,865 to $7,566. 7 In contrast, students in the lowest income quartile in both 
groups had, on average, from $9,031 to $10,259 in umnet need. This figure helps to illustrate 
the large differences in umnet need between the highest- and lowest-income students. MDRC’s 
analysis of program data from an experimental study of a financial aid program for traditional 
students at the University of New Mexico found that while 90 to 100 percent of students with 
umnet need of less than $9,000 in their first year registered in the second semester, college 
persistence rates declined for students with umnet need above $9,000. 8 This finding is in 
aligmnent with studies that suggest that low-income students who have significant financial 
needs often must work over 20 hours per week to support themselves and are more likely to 
extend their time to degree completion or even leave college altogether than if they worked 
less. 9 These trends collectively signal that for many lower-income students, the receipt of 
needed financial aid may be a critical factor in staying enrolled in college. 

The next section of this paper provides a review of the literature on the relationships 
between umnet financial need and progress toward degree completion as well as a review of 
the findings from studies that show the effects of financial aid intervention on student persis- 
tence and completion. The following section then examines MDRC’s experimental data from 
the Performance-Based Scholarship (PBS) Demonstration, a study spanning six states around 
the United States. This data analysis identifies important student characteristics and relation- 
ships between students’ financial aid and their persistence and academic achievement. 
Finally, the paper concludes with recommendations for how these collective findings can be 
utilized by each sector in the financial aid policy arena. Ultimately, this paper offers several 
puzzle pieces that fit together to help identify the students for whom college may not be 
affordable. It also offers recommendations for how policymakers in various sectors can 


6 Unmet financial need is the amount of financial need that remains after all forms of financial aid and the 
expected family contribution are subtracted from the total cost of attendance. 

7 Long and Riley (2007). 

8 For more details on the program in New Mexico see Miller, Binder, Harris, and Krause (2011). 

9 Choitz and Reimherr (2013); Cochrane and Hernandez-Gravelle (2007). 
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respond to the affordability issues faced by the low-income students they serve and want to 
help to achieve success in college. 


II. Financial Need, Financial Aid, and College Success 

This section provides an overview of the relevant research on financial need, aid, and college 
success from published and conference papers from the early 2000s to the present. This time 
frame was characterized by an exponential rise in college prices. 10 The discussion of the 
literature in this section is divided into two subsections: the first covers the relationship between 
student financial need and academic progress in college, and the second discusses the effects of 
financial aid on student academic success. 

Unmet Financial Need and Progress Toward Degree Completion 

Several studies have documented how high levels of unmet need may potentially relate 
to students’ prospects of staying enrolled in and completing college. Though the findings from 
these studies are primarily descriptive and correlational, they draw attention to issues that low- 
income students face. In Herzog’s statistical analysis of first-time, first-year students enrolled in 
a public four-year university from 1996 to 2002, he found that students with $1,000 in umnet 
need had dropout and transfer-out odds 7 to 10 percent above those of students with no unmet 
need. 11 An analysis of the National Postsecondary Aid Study (NPSAS: 04) shows that low- 
income, first-generation students’ mean amount of umnet need was nearly $6,000 before loans 
— half of their median annual income of $12,100. In an effort to meet their financial obligations 
in the absence of additional aid, 37 percent of these students worked full time outside of their 
studies. 12 King (2002) found that working 15 hours per week or more was negatively associated 
with college persistence and full-time enrollment. 13 In her study of trends in labor supply among 
undergraduates, Scott-Clayton notes that several studies have documented that working too 
many hours while in college may adversely affect student performance and progress toward 
degree completion. 14 The Bound, Lovenheim, and Turner study of student work behaviors also 
found that as work hours increased for undergraduate students, the length of time it took them to 


1 ''According to the College Board, “Published prices (both tuition and fees and room and board) at public 
four-year institutions rose more rapidly [between 2002-2003 and 2012-2013] ... than over either of the two 
preceding decades.” College Board (2012). 

"Herzog (2005). 

12 Engle and Tinto (2008). 

13 As cited in Cochrane and Hernandez-Grayelle (2007). 

14 Scott-Clayton (2012). 
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complete their degrees also increased. Additionally, Bound, Lovenheim, and Turner highlight 
evidence showing that students are likely to work more in order to meet rising college costs. 15 

Several researchers have analyzed the National Educational Longitudinal Survey 
(NELS: 88/2000) and reached similar conclusions about the low likelihood of college success 
when students attend college part time. The Attewell, Heil, and Reisel study showed that at 
four-year institutions, students who attended college part time during their first semester were 
about 5 to 7 percentage points less likely to graduate in 8.5 years than students who began 
college full time. At two-year institutions, students who attended part time during their first 
semester were about 8 to 13 percentage points less likely to earn an associate’s degree or higher 
within 8.5 years. The negative relationship between starting college with a lower course load 
and likelihood of graduation also appears to be greater for lower-income students. 16 

After analyzing the same NELS data, Adelman found that earning less than 20 credits 
by the end of the first year decreased the likelihood of degree completion, as did part-time 
enrollment generally. 17 Out of this analysis and his other studies, Adelman developed the 
“momentum theory,” which offers three ways that maintaining higher levels of credit enroll- 
ment may help students complete college. First, the intensity of enrollment at the start of college 
may set a trajectory for the likelihood of degree completion. Second, early momentum may 
have a stronger association with degree attainment than students’ sociodemographic back- 
ground and high school academic preparation. Third, maintaining momentum in college 
enrollment during summer and other low-enrollment terms may improve the prospects of 
degree completion. 18 This part of Adelman’s theory is rooted in his finding that summer 
attendance significantly increased the probability that a community college student would 
transfer to a four-year institution, by nearly 20 percentage points. 19 

In alignment with “momentum theory,” other studies show that full-time and summer 
college enrollment may increase the likelihood of success in college. Results from McComiick 
and Carroll’s study of first-year students in a four-year institution show that 91 percent of 
students who completed the year with 30 credits completed their degrees, while only 45 percent 
of students who enrolled in fewer than 20 credits completed their degrees. 20 O’Toole, Stratton, 
and Wetzel’s analysis of data from the Beginning Postsecondary Students Longitudinal Study 
(BPS: 90/94) found that students who always attended full time were much more likely to 


15 Bound, Lovenheim, and Turner (2010). 

16 Attewell, Heil, and Reisel (2012). 

17 Adelman (2006). 

18 Attewell, Heil, and Reisel (2012). 

19 Adelman (2005). This increase was observed at each of three levels of summer credits earned: zero cred- 
its earned, one to four credits earned, and more than four credits earned. 

20 As cited by Moore and Shulock (2009). 
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graduate within five years than students who attended part time for at least one term. 21 Attewell, 
Heil, and Reisel’s study using NELS:88/2000 data revealed that at four-year institutions, 
students who enrolled in courses during the summer after freshman year were significantly 
more likely to graduate, by about 4 to 1 1 percentage points, than students who did not attend 
college during the summer. Among two-year entrants who attended college during their first 
summer, the likelihood of graduating within 8.5 years increased from about 7 percent to nearly 
16 percent. 22 Finally, Kuh’s analysis of data from the National Survey of Student Engagement 
rendered results showing that full-time students are more engaged in college than their part-time 
counterparts. 23 Although these findings do not confinn causation, they collectively suggest a 
common pattern: students who enroll part time (to work, to lower the cost of their education in 
the short term, or for other reasons) are less likely to complete college. 

The Effect of Financial Aid on Academic Success 

While there is a lot of descriptive research on umnet need and college completion, firm 
knowledge about how financial aid affects students’ success in college is quite limited. Most 
studies that examine the relationship between financial aid and academic success offer correla- 
tional findings, while only a few studies use methods that yield true causal results. Correlational 
analyses of financial aid are unable to differentiate between the effects of additional aid and the 
influence of other student characteristics that are also associated with students’ likelihood of 
receiving aid. For example, when compared with other students, low-income students may both 
be more academically underprepared and have higher levels of umnet need. Because students 
vary in both regards, a correlational analysis will not be able to estimate the effects of additional 
aid alone, distinct from the effects of academic underpreparedness or other similar characteris- 
tics. 24 This subsection of the review focuses on studies that attempt to estimate the impact of 
financial aid on college success. The research methods used in the studies reviewed include 
randomized controlled trials (RCTs) that provide strong evidence for the causal impact of 
financial aid and quasi-experimental designs (such as difference-in-differences and regression 
discontinuity) that generally provide weaker evidence, but do more to account for confounding 
factors than correlational analyses. (See Table 1 for details on each of the studies discussed in 
this section.) 


' O’Toole, Stratton, and Wetzel (2003). 

22 Attewell, Heil, and Reisel (2012). 

23 Kuh (2003). 

In this scenario, students’ academic underpreparedness and any other characteristics that cannot be dis- 
entangled are said to be endogenous. Experimental designs allow researchers to estimate the exogenous effects 
of additional aid, that is, distinct from the influence of other student characteristics. 
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The literature on the impact of need-based grant aid on credits earned and college per- 
sistence has been consistently positive. Table 1 shows that Dynarski’s study of the effects of a 
change in a Social Security benefit found that by age 23, students who were eligible for this 
benefit were 16 percentage points more likely to complete one year of college. 25 Additionally, 
both Bettinger’s studies of the Ohio Opportunity Grant and the federal Pell Grant, as well as 
Goldrick-Rab, Harris, Kelchen, and Benson’s study of the Wisconsin Scholars Grant, found 
similar positive effects on retention after the first year of a grant aid increase of $ 1 ,000 to low- 
income students. 26 Goldrick-Rab, Harris, Kelchen, and Benson’s study also found positive 
effects on credits earned by the end of the first year. 27 

Grant aid, delivered either alone or accompanied by support services, has also had posi- 
tive effects on graduation rates. Table 1 also shows that in a study of the need-based Florida 
Student Access Grant, Long and Castleman found that eligibility for an additional grant of 
$1,300 not only had a positive effect on persistence, but also yielded an impact on graduation 
within six years from a public four-year postsecondary institution. 28 Strong positive effects on 
persistence and graduation were also found in a study where additional financial support 
covering tuition, fees, books, and transportation was accompanied by additional student ser- 
vices. 29 The Accelerated Study in Associate Programs (ASAP) was designed to accelerate 
students’ progress through community college by offering comprehensive financial support 
(such as tuition waiver covering the gap between aid and financial need, a public transportation 
pass, and free use of textbooks) and additional services (such as enhanced advising, career 
services, and special seminars) for three consecutive years. In addition to the positive effects on 
persistence in every academic term, including the strongest persistence effects during the 
summer and winter terms, ASAP had a positive effect on two-year associate’s degree comple- 
tion. The results from these two studies show that additional grant aid or similar financial 
support can help improve graduation outcomes. In the case of ASAP, the comprehensiveness of 
the additional services and the extended duration of the intervention may have played a substan- 
tial role in students’ success. (See Table 1 for details on the impacts of the ASAP program.) 30 

The results from studies of the effects of additional incentive-based grant aid on student 
success have been more mixed. 31 Incentive-based grants are often need-based grants that are 


25 Dynarski (2003). 

26 Bettinger (2004); Bettinger (2010); Goldrick-Rab, Harris, Kelchen, and Benson (2012). 

27 Goldrick-Rab, Harris, Kelchen, and Benson (2012). 

28 Long and Castleman (2012). 

29 Students continued to have some unmet need, as additional aid did not cover the total cost of attendance. 
30 Scrivener and Weiss (2013). 

3l Richburg-Hayes and Patel (2013). This paper examined nine studies that used experimental methods. 
The pooled effects were positive and statistically significant for some outcomes, like credits earned, but 
statistically insignificant for others, like persistence. 
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disbursed to students contingent on their meeting academic or other benchmarks. Both the 
Opening Doors study of performance-based scholarships in Louisiana (the study that inspired 
the development of the PBS Demonstration, findings from which will be discussed in Section 
III) and the Foundations for Success project found positive effects of a $1,000 aid increase on 
student persistence. 32 However, neither the Project STAR Study nor the University of Amster- 
dam program, both of which provided similar assistance, found any effects on credits earned or 
persistence. 33 (See Table 1 for details on the impacts found in these studies.) 

Though the correlational findings are interesting and the impact results are encouraging, 
additional research is needed to determine more clearly what kinds of aid interventions will help 
students the most and for which students they work best. Although the momentum perspective 
encourages summer enrollment, few studies have sought to isolate the effects of summer 
enrollment on persistence and graduation or the effects of summer financial aid on these two 
important outcomes. Though some of MDRC’s financial aid studies have yielded promising 
findings regarding the effects of aid on summer enrollment and credits earned, more research is 
needed to clarify how financial aid and summer enrollment relate to each other and to student 
success. Additionally, more targeted research could be conducted on how aid affects students’ 
academic outcomes at different levels of umnet need. This kind of research may help to identify 
the groups who may need additional aid the most in order to stay enrolled in college. Particular- 
ly in two-year institutional settings, where there is generally limited knowledge about what 
works to help students succeed, the field could greatly benefit from further research on how 
additional aid could encourage full-time enrollment (particularly among students who otherwise 
might have to work more hours) and ultimately help improve college completion rates. 

The next section of this paper focuses on MDRC’s analyses of experimental data from 
the PBS Demonstration. The discussion offers further insights into how low-income students’ 
academic performance relates to their financial aid status and how this particular type of aid has 
helped students make academic progress. 


III. Another Look at the Performance-Based 
Scholarship Demonstration 

This section uses experimental data from the PBS Demonstration to further explore some of 
the ways that financial aid may help students make more efficient progress toward degree 
completion. 


’’The Foundations for Success project studied the effects of a three-semester $750 fellowship (or $2,250 
in total) that gave students an incentive to attend 12 sessions of student success services. Richburg-Hayes et al. 
(2009); MacDonald et al. (2009). 

33 Angrist, Lang, and Oreopoulous (2009); Leuven, Oosterbeek, and van der Klaauw (2010). 
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The performance-based scholarships in the PBS Demonstration offered students an op- 
portunity to earn additional grant aid contingent on meeting an academic benchmark, such as 
earning a certain number of credits with a grade of “C” or better. Scholarship money was paid 
directly to students who met the benchmarks, and could be spent on whichever expenses the 
students incurred. Maximum annual award amounts varied from $1,800 to $4,500, depending 
on the site. (See Table 2 for a summary of the sites involved in the PBS Demonstration and the 
features of the program at each site.) The awards were not intended to fully meet students’ 
unmet need, which was often a much greater amount (as discussed below), but rather were 
meant to help improve their financial situation and introduce additional incentives for them to 
meet performance benchmarks. 

The studies conducted in the PBS Demonstration employed a random assignment 
methodology to evaluate the effects of offering performance-based scholarships to low-income 
students. Students who agreed to participate in each of the studies were divided randomly into 
two groups at each site: a “control group,” which received the normal financial aid and student 
support services offered to all students, and a “program group,” which received the opportunity 
to earn a performance-based scholarship in addition to the normal financial aid and other 
standard services offered. 34 Because students at each site were randomly assigned into the two 
groups, the groups are comparable both with regard to observable characteristics such as age, 
gender, and race, and with regard to unobservable characteristics such as tenacity and motiva- 
tion. Differences in outcomes can therefore be attributed to the effects of the program and 
causal relationships can be established. Due to its rigor, random assignment is known as the 
“gold standard” of program evaluation. 

Below, the characteristics of students who participated in some of the studies are de- 
scribed. Next, average enrollment outcomes for students in the control group are examined, and 
the relationship between financial aid and academic outcomes for these same students is 
explored. Lastly, the randomized design of these studies is used to estimate causal relationships 
between perfomiance-based scholarships and academic progress. 

Student Demographic and Financial Aid Characteristics 

The PBS Demonstration randomly assigned more than 12,000 low-income students at 
sites in six states in order to estimate the effect of the offer of the scholarship on their academic 
achievement. PBS Demonstration program models varied somewhat among sites, depending on 
the needs and student population of each institution, but the essential features of the scholarship 
were the same, as shown in Table 2. Table 3 summarizes some demographic and financial aid 


34 At some sites, the program group also received additional student services along with the opportunity to 
earn a performance-based scholarship. See Table 2 for details. 



characteristics at four PBS Demonstration sites for which detailed student-level financial aid 
data were available. 35 In order to show financial aid characteristics in the absence of the perfor- 
mance-based scholarship awards, only data for students assigned to the control group are 
shown, resulting in a total sample size of 1,702 across the four sites. Data for students in the 
program group are not shown, as these students’ financial aid packages were affected by their 
participation in the program. 

Table 3 shows that the demographic characteristics of the study sample varied among 
the sites. This is due partly to variation in the student populations at each school, and partly to 
the ways different PBS Demonstration programs were targeted. The program at Owens Com- 
munity College, for example, targeted students with children, while the program at Pima 
Community College targeted Hispanic males. The study sample at the University of New 
Mexico (UNM) was composed of more “traditional” students (younger, unmarried, and without 
children) than the other sites; this may be linked to the fact that UNM is a four-year institution 
while the other sites are community colleges. 

While the PBS Demonstration programs differ in some respects, all of the programs 
were targeted at low- income students. During their first year, 36 approximately 50 to 100 percent 
of control group students at each site had an expected family contribution (EFC) of zero, 37 and 
an average unmet need of approximately $4,000 to $8,600, representing anywhere from 
approximately 30 to 55 percent of the control group’s average cost of attendance at each 
institution. As shown in Table 3, however, there was some variability in students’ financial aid 
characteristics. For example, some students had much higher levels of unmet need, up to two to 


35 Detailed student-level financial data were available for the following PBS Demonstration sites: Hills- 
borough Community College in Florida, Owens Community College in Ohio, Pima Community College in 
Arizona, and the University of New Mexico. Student-level financial aid data available for each of these sites 
included at least one of the following: cost of attendance, EFC, gross need, or unmet need. 

36 Throughout this paper, “first year” generally refers to students’ first year in the study, beginning with the 
semester in which they were randomly assigned. However, the first-year financial aid values used in Tables 3 
and 5 do not all begin with students’ semester of random assignment due to variations in the ways that different 
institutions calculate annual financial aid values. (For example, some institutions calculate annual aid using a 
leading summer schedule, grouping aid in summer semesters with the following fall and spring semesters. 
Other institutions calculate annual financial aid using a trailing summer schedule, grouping aid in summer 
semesters with the preceding spring and fall semesters.) The first-year financial aid values used in Tables 3 and 
5 depend on the annual or semester-by-semester financial aid data available from each institution. When 
calculating annual aid awarded, researchers should be careful to align their calculations with the disbursement 
schedule used by each school when appropriate. 

37 EFC is an index that measures how much a student’s family is expected to contribute toward college 
expenses. It is calculated from a student’s Free Application for Federal Student Aid (FAFSA) and is based on 
family income, financial dependence status, and other factors. EFC is used as a criterion for awarding Pell 
grants and other federal aid. 
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five times the average amounts. Similarly, some students received much more financial aid than 
the average amounts. 

Academic and Financial Aid Outcomes 

As noted earlier, prior research has found that low-income students are an academically 
at-risk population. 38 Many students in the PBS Demonstration control group also exhibit 
additional at-risk characteristics. 39 This section examines the outcomes of students in the control 
group only (that is, not including students eligible for the performance-based scholarships) in 
order to show how this population performed in the absence of the intervention. The first set of 
analyses examines college persistence outcomes, while the second set examines the correlation- 
al relationships between financial aid and academic outcomes. 

An analysis of the persistence of students in the PBS Demonstration control group, 
shown in Table 4, found that these students were less likely to persist with the passage of each 
additional semester. 40 By their third semester in the study at Hillsborough Community College 
(HCC), fewer than 60 percent of HCC control group students were enrolled, and only about half 
as many were enrolled full time (that is, enrolled in 12 or more credits). Similar results are 
observed for students’ first four semesters of the study at Owens, and first two semesters of the 
study at Pima. Students at UNM were somewhat more likely to remain enrolled, with almost 75 
percent of control students enrolled in their fourth semester of the study, and almost 70 percent 
of the control group enrolled full time. 

Students in the control group who received larger amounts of aid during their first year 
were somewhat more likely to persist in college, and also attempted and earned a greater 
number of credits in subsequent semesters, on average. Though there are some nuances, the 
relationship of these outcomes for low-income students to the amount of aid they received is 
similar to some findings in the literature. 41 The size of the relationship observed at PBS Demon- 
stration sites varies, but on average, an additional $1,000 in aid received is associated with a 2 to 
7 percent increase in persistence, and approximately 0.2 to 1 .2 additional credits attempted or 


38 Titus (2006); Attewell, Heil, and Reisel (2012). 

39 Prior research found that seven “risk factors” were associated with a higher likelihood of leaving post- 
secondary education without attaining a credential. These risk factors included characteristics such as having 
children, being a single parent, and being considered financially independent for FAFSA purposes. See Flom, 
Berger, and Carroll (2004). 

40 Throughout the tables and figures in this paper, spring and summer semesters are combined when calcu- 
lating semester-by-semester student academic outcomes such as persistence, credits attempted, and credits 
earned. Because student enrollment in summer semesters is typically lower than enrollment in the fall or 
spring, combined spring and summer academic outcomes are similar to outcomes for the fall or spring 
semesters alone. 

41 See Gross, Flossier, and Ziskin (2007). 
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earned. Across all four sites, these relationships are statistically significant at the 1 percent level, 
although the magnitudes of the relationships are small. 42 The top half of Table 5 shows these 
results. The bottom half of Table 5 shows the results of a similar analysis of the relationships 
between umnet need and persistence, umnet need and credits attempted, and unmet need and 
credits earned. These relationships are less consistent. While an additional $1,000 of umnet 
need is associated with a 2 to 4 percent decrease in persistence and a decrease of approximately 
0.2 to 0.8 fewer credits attempted and earned at two of the four PBS Demonstration sites (HCC 
and UNM, statistically significant at the 1 percent confidence level), it is not associated with 
changes in these same academic progress outcomes at Pima. At Owens, an additional $1,000 of 
umnet need is associated with a 1 percent decrease in persistence, but not with changes in 
credits attempted or earned. 

In addition to the results shown in Table 5, other control group student baseline demo- 
graphic characteristics such as gender and first-generation student status were tested for rela- 
tionships with students’ chances of persistence. However, these other characteristics did not 
show any relationship with persistence for the control group students in the sample. 43 

Another analysis of data for the control group was conducted to see if decreases in per- 
sistence could be linked to particular levels of umnet need. Data at one site, UNM, revealed that 
student persistence drops for levels of unmet need above $9,000. Figure 2 illustrates these 
results. The x-axis shows students’ unmet need in their first year in the study, while the y-axis 
shows the percentage of students who enrolled in the second semester. Students with an umnet 
need of less than $9,000 enrolled at a rate of 90 to 100 percent regardless of their level of umnet 
need. However, students with higher levels of unmet need were less likely to persist. 44 While 


42 The level of statistical significance is often used to assess the strength of statistical results, and the likeli- 
hood that those results are true results, as opposed to a product of random noise in the data. A 1 percent level of 
significance indicates that there is only a 1 percent chance that such results reflect random noise, and would be 
observed in the absence of any real relationship between the amount of financial aid received and persis- 
tence/credits attempted/credits earned. 

43 The following student characteristics were examined to see if they correlated with persistence: age, gen- 
der, current employment status, mother’s level of education, father’s level of education, EFC in the student’s 
first year, a proxy for whether the student was considered financially independent under FAFSA rules, whether 
the student was the first person in the family to attend college, highest high school grade previously completed, 
and whether the student’s primary language was English. Correlations were estimated for each site separately. 
Some characteristics appeared to be significantly correlated with persistence for selected sites, but no character- 
istics were consistently correlated with persistence. These correlations were estimated using data for control 
group students, but similar results could be expected for all students in the sample. This is because random 
assignment generates control and program groups that are comparable in both observed and unobserved 
characteristics. 

44 Similar data for other sites were also analyzed, but they did not show the same pattern as was seen at 
UNM. At Pima, F1CC, and Owens, increased levels of unmet need were associated with lower levels of 
persistence, but no sharp drops in persistence were found at particular levels of unmet need. 
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any potential causal relationships are difficult to establish, these results suggest that the declin- 
ing levels of persistence among UNM control group students shown in Table 2 may be driven 
by students with an umnet need in excess of $9,000. 

Though Table 5 shows that control group students’ levels of financial aid and umnet 
need were correlated with both their persistence and academic achievement, any potential 
causal relationships are difficult to determine using data for the control group alone. While 
increased financial aid or lower umnet need may lead to higher levels of academic achievement, 
it also is likely to be the case that students who emoll in more classes receive greater annual 
amounts of financial aid. Therefore, it is plausible that higher amounts of aid received or lower 
levels of unmet need do not cause increased persistence or academic achievement; rather, these 
relationships could be driven by a third factor or set of factors. 

Experimental Analysis 

The PBS Demonstration randomly assigned students to be offered an incentive-based 
scholarship. This section of the analysis makes use of that design in order to estimate causal 
relationships. This analysis first examines the effect of performance-based scholarships on 
students’ total financial aid received in their first year in the program. 45 Figure 3 shows these 
effects for four PBS Demonstration sites for which these results are available. Among these four 
sites, the program increased total financial aid received by an average of around $500 to $2,000, 
which ranges from 7 to 38 percent of the aid that students would have received without the 
program. Excluding the New York impacts, which are the largest, the impact on total financial 
aid received at the other three sites ranges from 7 to 25 percent of the aid that would have been 
received in the absence of the performance-based scholarships. While impacts on umnet need 
are not available, it can be hypothesized that the increases in aid were probably accompanied by 
corresponding decreases in students’ unmet need. 

In addition to impacts on financial aid received, performance-based scholarships had 
impacts on enro llm ent outcomes and credit accumulation at several sites. At the Ohio, New 
York, and Arizona sites, the program led to a greater proportion of students enrolling full time 
in certain semesters: the size of the effect ranged from around 6 percent of students in Ohio and 
New York to 13 percent of students at Pima Community College in Arizona. 46 At the New York 


45 lt is important to note that because the program offered students the opportunity to earn additional aid 
but only disbursed the aid to students who met the relevant benchmarks, this analysis estimates the impact of 
the offer of aid rather than the receipt of the aid itself. The offer of aid rather than the receipt of aid can be 
understood in terms of intent-to-treat (ITT) and treatment-on-the-treated (TOT) estimates, respectively. See 
Bloom (1984). This analysis uses ITT estimates. 

46 See Table 5 of Cha and Patel (2010); Appendix Table B.l of Patel and Rudd (2012); and Table 4.1 of 
Patel and Valenzuela (2013). 
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site, the program also led to a greater proportion of students enrolling in the summer semester, 
again with an impact of around 6 to 7 percent. 47 Despite effects on other enrollment outcomes, 
the program did not have impacts on overall levels of registration. Perfonnance-based scholar- 
ships also led to an increase in credit accumulation at multiple sites, as shown in Figure 4. 48 
Credit impacts ranged from around one to around three credits. 49 

A three-year follow-up analysis has also been conducted at the three PBS Demonstra- 
tion sites in Ohio. Three years after beginning the program, the program group completed 
degrees at a higher rate than the control group. While 23.3 percent of control group students 
earned degrees, 26.9 percent of the program group earned degrees. This 3.5 percentage-point 
difference was statistically significant at the 10 percent level. This finding provides evidence 
that the perfonnance-based scholarships in Ohio helped program group students earn degrees 
who would not have done so in the same time frame in the absence of the intervention, though 
additional follow-up is needed to determine whether the effects last. 50 

Discussion 

The findings above suggest that perfonnance-based scholarships may have improved 
the affordability of college for program group students in several ways. In the short term, the 
program increased the amount of aid that students received and may have reduced their umnet 
need. While the program did not have an effect on students’ overall persistence in college, it 
did convince students at several sites to enroll full time and invest their time and money more 
heavily in college in the short term. 51 In the longer term, performance-based scholarships may 
have increased the affordability of college by shortening some students’ expected time to 
degree completion, lowering the average number of semesters that they would need to enroll 
before receiving a degree, and therefore lowering their total cost for obtaining a degree. 52 At 


47 See Table 3.2 of Patel and Rudd (2012). 

48 The sites shown include Louisiana; Louisiana was part of the earlier Opening Doors Demonstration that 
also followed a similar performance-based scholarship model and that served as the basis for the PBS Demon- 
stration. Figure 4 shows that there were no impacts on credits earned or enrollment in Florida. This may be due 
to the design of the Florida performance -based scholarships, which unlike other performance-based scholar- 
ships focused specifically on students’ progress through developmental math, not their overall level of 
enrollment or academic achievement. 

49 For a cross-site summary of the effects of performance-based scholarships, please refer to Patel, 
Richburg-Hayes, de la Campa, and Rudd (2013). 

50 See Table 2 ofPatel, Richburg-Flayes, de la Campa, and Rudd (2013). 

5 'Students who enroll full time may face higher tuition costs; full-time enrollment therefore represents an 
additional investment of both time and money. 

52 lt is also possible that performance-based scholarships did not shorten students’ average time to their 
degrees: because not all credits can be applied to degrees, an impact on credits earned may not necessarily 
translate to students earning a greater number of degree-applicable credits. 
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the Ohio sites, the program resulted in a higher proportion of students receiving a degree after 
three years. Through impacts on both financial and academic outcomes, performance-based 
scholarships may have improved multiple aspects of affordability for students participating in 
the program. 

The magnitude of the effects observed for performance-based scholarships is broadly 
similar to that of the correlations shown in Table 5, which show that additional financial aid is 
correlated with slight improvements in college persistence, credits attempted, and credits 
earned. However, the experimental performance-based scholarships results offer two distinct 
advantages over this prior analysis. First, the results are based on a randomized controlled trial, 
which provides a strong argument for the causal effects of additional, incentive-based aid on 
academic outcomes. Second, because the program model does not award scholarships to 
students unless they meet academic benchmarks, the program as a whole may have lower costs 
than a program that distributes grants to all students, regardless of their academic progress. 
Depending on the size of the effects observed, the size of the awards distributed, and the 
frequency with which the awards are distributed, performance-based scholarships may be more 
cost-effective than a program model that awards grants to students without requiring them to 
meet performance benchmarks. 

Performance-based scholarships’ positive impacts are encouraging, given that the 
program increased students’ financial aid by around $500 to $2,000 per year, a relatively 
small amount compared with students’ full cost of attendance. Taken together with the 
positive impacts, the size of the performance-based scholarships suggests that financial aid 
reform could come in the form of small increases to existing sources of aid. The incremental 
nature of performance-based scholarships could help refine the ways in which small amounts 
of funding are targeted and the conditions under which the aid is disbursed. The analyses 
above suggest several financial and academic outcomes that could be useful foci for future 
research and reform efforts: 

• Improved targeting of aid. Research on performance-based scholarships has 
found that the scholarships are effective for a variety of types of students (see 
Table 6, which shows pooled impacts on credits earned after one year for all 
PBS Demonstration sites except California). 53 It is important to note that the 
PBS Demonstration targeted low-income students. However, additional work 


53 While the impacts shown in Table 6 are statistically significant, they are not different for different stu- 
dent subgroups, with the exception of gender subgroups. That is. Table 6 shows that men do better in the 
program group and that the intervention has more effect for men than women. However, for other subgroups 
tested (such as Hispanics compared with non-Hispanics), while both subgroups do well in the program group, 
the intervention does not work better for one group than the other. 
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could be done to identify those student populations — within the low-income 
student populations studied in the PBS Demonstration — who are least likely 
to persist in college and for whom college may be least affordable. Scholar- 
ships similar to those offered in the PBS Demonstration might be most effec- 
tive if targeted to such students. The analyses described here show that it can 
be difficult to strongly predict student success, at least when focusing on 
low-income students, so it may be necessary to collect new data in order to 
better target financial aid. At some institutions, such as UNM, there may also 
be particular levels of unmet need associated with sharply worse academic 
outcomes. It may be helpful to ensure that students remain below these criti- 
cal levels of umnet need. 

• Increasing intensity of enrollment. Results at PBS Demonstration sites in Ar- 
izona, New York, and Ohio suggest that small amounts of aid can be used to 
increase the proportion of students who choose to enroll full time (defined as 
enrolling in 12 or more credits). Perfonnance-based scholarship programs in 
Ohio and Arizona offered differential scholarship amounts in order to induce 
students to enroll full time, but the same was not true in New York. As dis- 
cussed earlier, previous research has found that full-time enrollment is corre- 
lated with a higher probability of academic success, and early results in Ohio 
suggest that the program’s impacts on full-time enrollment may be helping to 
drive impacts on graduation. 54 Future financial aid interventions could be 
structured to encourage full-time enrollment. 

• Enrollment in summer/winter terms. Results at PBS Demonstration sites in 
New York show that perfonnance-based scholarships can increase the pro- 
portion of students who enroll during summer sessions. As noted earlier, 
summer enrollment is correlated with higher transfer rates to four-year in- 
stitutions, so financial aid during the summer may be a potentially fruitful 
area for additional research. 55 In addition to performance-based scholar- 
ships, other types of interventions may also be able to increase summer en- 
rollment. For example, the ASAP program found a substantial impact on 
the rate of summer enrollment among program students, even though 
summer enrollment was a not a program requirement. 56 As noted earlier, 


54 Kuh 2003; O’Toole, Stratton, and Wetzel (2003); Adelman (2006); Attewell, Heil, and Reisel (2012). 
55 As noted earlier, Adelman (2006) found that earning credits during the summer significantly increased 
the probability of transfer to a four-year institution. 

56 See Figure 1 in Scrivener and Weiss (2013). 
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ASAP’s comprehensive array of services included a variety of both finan- 
cial and nonfinancial components . 57 


IV. Policy Recommendations 

The findings in this paper from descriptive, correlational, and causational analyses all contribute 
pieces to the puzzle of defining college affordability. Collectively, these findings show that 
student behaviors and characteristics — such as unmet need, high levels of employment while 
enrolled in college, college enrollment intensity (part time or full time), and student response to 
aid received — help identify how affordable college is for a particular student . 58 Moreover, the 
impact findings from financial aid interventions provide some clarity regarding how well 
different approaches to offering additional aid have worked in helping low-income students to 
progress in college and ultimately obtain a degree. Given these findings, how should financial 
aid policymakers and administrators respond? What are some strategies that can be further 
tested in each sector of the financial aid policy landscape? This section of the paper seeks to put 
the puzzle pieces together by offering recommendations for strategies that may help make 
college more affordable and, at the same time, may help to illustrate what it looks like for 
college to be affordable for students. 

Summer and Winter Financial Aid 

Offering aid to students during the summer or winter would be a prime opportunity to 
test if aid during short terms helps students make stronger progress toward degree completion. 
There are several ways that different policy actors could approach this goal. 

• The federal government: Test Pell or Supplemental Educational Opportunity 
Grant (SEOG) funding. Though there is currently no summer or winter fed- 
eral financial aid, this type of financial aid could be pilot tested in an experi- 
mental environment where students were randomly offered the opportunity 
to receive summer and winter grant aid. Summer Pell (also known as Year- 
Round Pell) was previously implemented, but was cut due to rising costs. Ty- 
ing the reintroduction of summer or winter Pell awards to some of the other 
strategies discussed in this section (targeted aid, for example, or incremental 
aid disbursements) could help control program costs and make the program 
more sustainable. Federal Student Aid could also partner with selected two- 


57 Program students received financial aid to cover all tuition and fees not already covered by federal and 
state aid. See Section II of this paper for additional description of ASAP program components. 

58 The relationship between high levels of employment and college affordability is discussed in the review 
of the literature in Section II. 
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year and four-year institutions to test offering additional SEOG funds for 
disbursement to students during summer and winter terms . 59 

• States and institutions: Utilize internal or external grant funding. States and 
institutions with flexible grant aid dollars could allocate some of those funds 
toward grants for summer and winter college enrollment. Additionally, states 
and institutions could seek out partnerships with local and national donor or- 
ganizations committed to helping low-income students graduate from col- 
lege. The effect of summer grant aid on students’ academic success could be 
tested by randomly assigning students to one of three groups: aid during the 
summer and winter, more aid during all academic terms, or no additional aid. 
Designing a test with these three variable conditions would help to inform 
the field about how much summer aid helped students, and about whether 
summer aid alone was enough to see a meaningful impact on student success. 

• Foundations: Invest in grant aid and research. Foundations can inform fed- 
eral, state, and institutional policy by funding scholarships and experimental 
research to leam how summer/winter aid affects students’ progress toward 
degree completion. 

Aid Aimed More Intentionally at Students Who May Need It Most 

Throughout the PBS Demonstration, MDRC worked with sites to help them deter- 
mine which students might be most in need of additional aid and why. Through this process, 
different low-income student populations were identified, and aid was targeted at groups who 
were deemed to potentially have some of the greatest needs at each site. Financial aid inter- 
ventions could continue the work of identifying and targeting aid to at-risk low-income 
student populations. 

• The federal government: Implement additional requirements or incentives. 
Federal Student Aid can require or provide incentives for states and institu- 
tions to focus more of their discretionary aid on students who national stu- 
dent financial aid data indicate may be at risk of not enrolling or of dropping 
out due to financial challenges. The federal government could also research 
ways to restructure the FAFSA in order to direct more aid to student popula- 
tions for whom it may have a larger impact. 


^Institutions have discretion over the schedule of SEOG disbursements, so this initiative would need to be 
a partnership between Federal Student Aid and selected institutions. 
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• States and institutions: Intensively explore in-house data. States and institu- 
tions may want to consider testing various fonns of targeting to leam how to 
most effectively allocate future increases or decreases in aid. By thoroughly 
examining their student data on how student outcomes correlate with levels 
of unmet need and other characteristics, they might identify particular at-risk 
groups to whom it might be good to target additional aid. 

• Foundations: Conduct in-depth review of public student data or institutional 
data. By using public student data from sources like the Integrated Postsec- 
ondary Education Data System (IPEDS), foundations could gain a better 
sense of which students in different geographic areas might have the greatest 
need for additional financial aid. Based on the results of this review of data, 
foundations could better target grant and scholarship aid dollars to students, 
institutions, or states. 

Comprehensive Aid 

After identifying the students who have the greatest needs, policymakers may find that 
in addition to offering these students supplemental financial aid, more services may be needed 
to help these students succeed, such as the kinds of assistance that were offered in the Founda- 
tions for Success, ASAP, and PBS Demonstration programs. 

• Institutions: Attach support for services to grant aid. Based on results from 
studies that examined the effects of programs that required or offered support 
services to students in tandem with financial aid or support, institutions could 
identify students who might benefit from receiving support services in addi- 
tion to aid and then pilot test one of the above-mentioned models that com- 
bines financial aid and support services. 

• Foundations: Support demonstrations of promising strategies. Foundations 
can help contribute to the body of evidence necessary to determine if com- 
prehensive programs have replicable effects by financially supporting pro- 
gram components, such as scholarships and book vouchers, and research into 
what aspects of comprehensive programs are most needed in order to achieve 
large positive effects on educational outcomes. 

Alternative Aid Distribution Models 

In order to encourage students to stay enrolled and to reduce the inefficiencies and risks 
of disbursing financial aid at the start of the semester in a lump sum, states and institutions 
could consider incremental aid disbursement options. 
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• States: Test performance-based aid against merit-based aid programs. States 
that primarily offer merit-based aid could test performance-based scholar- 
ships as an alternate approach to allocating scholarship dollars. This ap- 
proach might not only increase equity among lower- and higher-income stu- 
dents being offered aid, but might also increase the efficiency of the grant 
program by only allocating dollars to students after they met program re- 
quirements during the temi that the aid was offered, rather than disbursing 
aid based on perfomiance prior to college. This might help to stretch grant 
aid further and reach more students while providing students with a stronger 
incentive to rise to and maintain strong academic progress. Additionally, 
states that are considering increasing or decreasing financial aid may want to 
consider connecting these changes to a perfomiance-based scholarship or 
grant. 

• Institutions: Allocate lump sum aid in small increments. Altering the way 
loan dollars are disbursed to students may both increase the likelihood of stu- 
dents staying enrolled and make it more likely for the money to be used in 
the way that it was intended. If loan funds were disbursed to students in small 
biweekly or monthly increments, those students could potentially better 
budget and manage the financial aid that they receive. This type of strategy 
could easily be tested by offering the new and old approach to disbursing aid 
to students at random . 60 Some students receive large financial aid refunds and 
then drop out of college, both putting themselves at risk of not being able to 
pay back their student loans and also contributing to lower levels of efficien- 
cy in the aid program itself. This approach might therefore help to make col- 
lege more affordable to students in both the short and longer term. 

Research and Refinements of Work-Study to Help Improve College 

Success Among Low-Income Students 

Given that students who are employed full time while enrolled in college risk dropping 
out or at least prolonging their time to degree completion, work-study could be expanded to 
more low- income students to reduce their need to work full-time jobs disconnected from their 
educational pursuits. 

• The federal government: Test against the current model a modified work- 
study program designed to help low-income students make career advances 


60 MDRC is currently evaluating one program, Aid Like A Paycheck (ALAP), that is designed to apply 
this approach to some grant awards. 
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while in college. To date, little research has been conducted to test the effec- 
tiveness of the Federal Work-Study program. The few studies that have been 
conducted of such aid have been quasi-experimental and have yielded heter- 
ogeneous findings . 61 Given the amount of money expended on this aid pro- 
gram, it would be a worthy endeavor to conduct experimental research on 
both the current model and a refined model designed to address the challeng- 
es raised in this paper, to clarify how to help students with work-study aid 
most effectively. 

Conclusion 

The consistent thread comiecting all of these recommendations is that they all make de- 
fining college affordability an active rather than a passive process. Much can be learned about 
college affordability by taking steps to help students stay enrolled and succeed academically. 
Testing strategies as pilot programs to determine their effectiveness in helping students make 
stronger academic progress is the key to moving toward policies that actually work. By coordi- 
nating efforts across policy sectors to implement an innovative agenda and strategy within the 
current level of resources, testing and refinement can be implemented that could shed light on 
what is meant by “affordable” while simultaneously helping students. 


61 Scott-Clayton (2011). 
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Table 1 


Studies Examining the Effect of Financial Aid on Academic Success 


Key Study 




Eligibility 

Analysis 

Maximum 

Program 

Study 

Methodology 

Criteria 

Samule 

Amount 

Duration 

Social Security 

Difference-in- 

Child under 23 with 

3,545 

N/A 

5 years 

Student Benefit 

differences 

a deceased parent 

(P: 2,558; 



Program 
(United States) 
Dynarski, 2003 


Full-time student 

C: 987) 





High school senior 
in 1979, 1980, 1981, 
1982, or 1983 




Pell Grant Ohio 

Regression 

First-time, first-year 

37,028 

$3,125-3,300 N/A 

(United States) 

discontinuity 

student in the 1999- 




Bettinger, 2004 


2000 academic year 






Enrolled in an Ohio 
public 2-year or 4- 
year institution 




Opening Doors 

Random 

Age 18 to 34 

537 

$2,000 

2 semesters 

Louisiana 

assignment 


(P: 264; 



(United States) 
Richburg-Hayes 


Parent 

C: 273) 



et ah, 2009 


Family income 
below 200 percent 
of poverty level 





Impacts 

1. By age 23, students who were eligible for aid were 24.3 
percentage points more likely to attend college full time. 

2. By age 23, students who were eligible for aid were 16.1 
percentage points more likely to complete at least a year of 
college. 

3. Each $1,000 of aid offered increased completed 
schooling by 0.16 years. 


A $ 1 ,000 increase in Pell receipt improved first-year 
persistence by 2-4 percentage points. 


1. The study yielded a 3.2 percentage-point increase in 
retention per $ 1 ,000 of additional aid. 

2. On average, students earned an additional 1.5 credits in 
the first year and 1.7 credits in the second year per $1,000 
increase in aid. 

3. The program increased the likelihood of full-time 
enrollment in the first semester by 9.3 percentage points, 
in the second semester by 20.3 percentage points, and in 
the third semester by 10.7 percentage points. 



Table 1 (continued) 


Key Study 




Eligibility 

Analysis 

Maximum 

Program 


Study 

Methodology Criteria 

Sample 

Amount 

Duration 

Impacts 

Foundations for 

Random 

Enrolled full time 

3,110 

$2,250 

3 semesters 

An additional $1,000 of aid improved student persistence 

Success 

assignment 


(P: 2,056; 



by 2.3 percentage points. 

(Canada) 
MacDonald et 


Canadian citizen 

C: 1,054) 




al., 2009 


Identified as at-risk 
based on Accuplacer 
and FastTrack 
survey 





Project STAR 

Random 

Full-time, first-year 

1,571 

$5,000 

2 semesters 

The program had little effect on credit accumulation or 

(Canada) 

assignment 

college student 

(P: 619; 



persistence. 

Angrist, Lang 
and 


High school GPA in 

C: 952) 




Oreopoulous, 

2009 


first to third quartile a 





Ohio College 

Difference-in- 

First-time, incoming 

83,259 

$2,496 

N/A 

An $1,000 increase in aid decreased dropout rates by 1.7- 

Opportunity 

differences 

student in the 2005- 

(P: 



2.4 percentage points. 

Grant 


2006 or 2006-2007 

41,062; 




(United States) 


academic year 

C: 




Bettinger, 2010 


Defined as 
dependent by 
FAFSA 

42,197) 





Enrolled in a public 
Ohio 2- or 4-year 
institution 



Table 1 (continued) 


Key Study 

Eligibility Analysis Maximum Program 

Study Methodology Criteria Sample Amount Duration Impacts 


University of 

Random 

First-year college 

249 

$215 or 

$644 3 terms 

This program had little effect on credit accumulation or 

Amsterdam 

assignment 

student in economics 

(P: 167; 



persistence. 

(Netherlands) 

Leuven, 

Oosterbeek, and 
van der Klaauw, 
2010 


or business 

C: 82) 




Wisconsin 

Random 

Wisconsin resident 

14,847 

$3,500 

Up to 5 

1 . An increase of $ 1 ,000 in aid improved second-year 

Scholars Grant 

assignment 


(P: 2,125; 


years or 10 

retention by 2.5 percentage points. 

(United States) 


Attended a state 

C: 


academic 

2. Students who were enrolled in the program were 2.4 

Goldrick-Rab, 


public high school 

12,722) 


semesters 

percentage points more likely to earn at least 12 credits by 

2012 


Enrolled full time at 
a public state 
university 




the end of the first semester. 



Completed the 
FAFSA, qualified 
for a Pell Grant and 
still had some unmet 
need after all grant 
aid 







Table 1 (continued) 


Key Study 

Eligibility Analysis Maximum Program 

Study Methodology Criteria Sample Amount Duration 

Florida Student Regression- High school senior 7,171 $1,300 The grant 

Access Grant discontinuity in 2000-2001 could be 

(United States) renewed 

Long and Enrolled in a public from one 

Castleman, 2012 postsecondary year to the 

institution in Florida next 




EFC between $1,000 
above and below the 
grant eligibility 
cutoff of $1,590 


Accelerated 

Random 

Eligible for a Pell 896 

The balance 

Study in 

assignment 

Grant 

of student’s 

Associate 



tuition after 

Programs 


Needing to take 1 or 

other grant 

(United States) 


2 remedial classes 

aid was 

Scrivener and 
Weiss, 2013 


New York City 
resident 

applied 


New or returning 
student with less 
than 12 credits 


Impacts 


1. $1,000 in grant aid eligibility (in 2000 dollars) 
increased the likelihood of earning a bachelor’s degree 
within six years by 4.6 percentage points. 

2. Continuous enrollment through the spring semester in 
the first year increased by 4.3 percentage points per 

$ 1 ,000 in additional grant aid eligibility. 

3. On average, students eligible for $1,000 in aid earned 
2. 1 more credits in three years. 


1 . The program improved fall and spring semester 
retention by 8 to 10 percentage points. 

2. The program increased intersession enrollment by 10 to 
25 percentage points. 

3. After two years, the program increased the amount of 
credits earned by an average of 7.6 credits. 

4. The program increased two-year graduation rates by 5.7 
percentage points. 



SOURCES: Angrist, Lang, and Oreopoulous (2009); Bettinger (2004); Bettinger (2010); Dynarski (2003); Goldrick-Rab, Harris, Kelchen, 
and Benson (2012); Leuven, Oosterbeek, and van der Klaauw (2010); Long and Castleman (2012); MacDonald et al. (2009); Richburg- 
Hayes et al. (2009); Richburg-Hayes and Patel (2013); and Scrivener and Weiss (2013). 

NOTES: P = program group, C = control group. FAFSA = Free Application for Federal Student Aid. EFC = Expected Family Contribution, 
the amount of money that a family is expected to be able to contribute to a student’s education, as calculated according to federal guidelines. 
a Academic benchmark was conditional on high school grade quartiles. See Richburg-Hayes and Patel (2013) for description. 
b The low-reward group was offered 227 euros ($215 in 2002 dollars); the high-reward group was offered 681 euros ($644 in 2002 
dollars). 
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Table 2 


Summary of Performance-Based Scholarship Programs 


Opening Doors 
Louisiana 

PBS 

Arizona 

PBS 

California 

PBS 

Florida 

PBS 

New Mexico 

PBS 
New York 

PBS 

Ohio 

Sample selection criteria 







Age 

18-34 

No age criteria 

16-19 

At least 1 8 

17-20 

22-35 

At least 1 8 

Income 

requirements 

Family income 
below 200 

percent of 
poverty level 

EFC below 
5,273 a 

Below Cal Grant 
A/C Income 
Threshold 11 

EFC below 
5,273 a 

Pell-eligible 

Pell-eligible 

Zero EFC a 

Additional 

requirements 

Parent earned 
a high school 
diploma, GED 
certificate, or 
passing score on 
college entrance 
exam 

Hispanic male 
< 45 credits earned 

1 st-year student 
Completed 
FAFSA and Cal 
Grant 

In need of 
remedial 
math 

1 st-year student 

Live away from 
parents 
In need of at 
least 1 remedial 
course 

Parent 

Intervention 








Scholarship 
amount per term 

$1,000 

Up to $1,500 

$333-1,000° 

$600 

$1,000 

$1,300 

$300-900 

Scholarship 

duration 

2 semesters 

3 semesters 

1-4 semesters 

3 semesters plus 
1 summer term 

4 semesters 

2 semesters plus 1 
summer semester 1 

2 semesters 
or 3 quarters 

Max amount 

$2,000 

$4,500 

$1,000-4,000° 

$1,800 

$4,000 

$2,600-3,900 d 

$1,800 

Sample size 
Program 
Control 

537 e 

264 

273 

1,028 

611 

417 

4,92 l f 
1,640 
3,281 

1,075 

674 

401 

1,081 

536 

545 

1,502 

754 

748 

2,285 

1,359 

926 


(continued) 




Table 2 (continued) 


Opening Doors 
Louisiana 

PBS 

Arizona 

PBS 

California 

PBS 

Florida 

PBS 
New Mexico 

PBS 
New York 

PBS 

Ohio 

Performance benchmarks 







Academic 

criteria 

Enroll in college, 
and complete at 
least 6 credits 
with a “C” 
average or 
better 

Part time: 6-11 
credits with “C” or 
better in each 
Full time: 12 or more 
credits, “C” or 
better in each 

Enroll in college, 
and complete at 
least 6 credits 
with a “C” 
average or 
better 

Complete a 
sequence of 
math courses 
with a “C” or 
better 

Complete at least 
12 (first semester) 
or 15 (subsequent 
semesters) credits 
with a “C” 
average or better 

Enroll in college, 
and complete at 
least 6 credits 
with a “C” or 
better in each 

Part time: 6-1 1 
credits with “C” 
or better in each 
Full time: 12 or 
more credits, “C” 
or better in each 

Service criteria 

Meet with 
adviser 

Meet with adviser, 
complete tutoring 
and workshop 
requirements 

N/A 

Complete 

tutoring 

requirements 

Meet with adviser 

N/A 

N/A 

ParticiDatins institutions 







Participating 

institutions 

Delgado 

Community 

College 

Louisiana 

Technical 

College 

Pima Community 
College 

California 
Student 
Aid Commission 
and L.A. 
Chamber of 
Commerce 

Flillsborough 

Community 

College 

University of New 
Mexico 

Elostos 

Community 

College 

Borough of 
Manhattan 
Community 
College 

Lorain County 
Community 
College 

Owens 

Community 

College 

Sinclair 

Community 

College 


(continued) 



Table 2 (continued) 


SOURCE: Performance-Based Scholarship study design. 

NOTES: Scholarships are limited to the institution where the evaluation occurred, with the exception of PBS California. For this site, scholarships were 
portable to any accredited, degree-granting, two-year or four-year postsecondary institution in the U.S. 

a The EFC (Expected Family Contribution) is the amount of money that a family is expected to be able to contribute to a student’s education, as calculated 
according to federal guidelines. Students with an EFC of up to 5,273 during the 2010-201 1 year were eligible for federal Pell grants. 

b Cal Grant is a financial aid program funded by the state of California. The awards do not have to be paid back, but to qualify students must fall below 
certain income and asset ceilings. 

c The study in California randomly assigned program group members to one of six scholarship types that varied in amount (from $1,000 total to $4,000 total) 
and duration (from one term to two years). Students could take the award to any degree-granting, accredited institution in the country, and payments were 
adjusted to reflect the institution type (quarter or semester). 

d The study in New York randomly assigned program group members to one of two scholarship types. One type was offered over two semesters only; the 
other was offered over two semesters plus one summer semester. 

‘Although there were 1,019 study participants, only 537 participants from the first and second cohorts were analyzed, as Flurricane Katrina disrupted the 
follow-up period for the third and fourth cohorts. 

‘Although there were 5,160 study participants, undocumented immigrant students were excluded from the analysis because of data reliability concerns. 

Thus, the analysis sample was 4,92 1 participants. 
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Table 3 


Demographic and Financial Characteristics of Control Group Students: 


Selected Performance-Based Scholarship Sites 


Characteristics 

Hillsborough 

Community 

College 

Owens 

Community 

College 

Pima 

Community 

College 

University 

of 

New Mexico 

Gender 3 (%) 

Male 

32.2 

15.0 

100.0 

39.8 

Female 

67.8 

85.0 

0.0 

60.2 

Age (%) 

17-26 years old 

58.9 

44.2 

74.3 

100.0 

27-30 years old 

12.0 

17.1 

6.5 

0.0 

3 1 and older 

29.2 

38.6 

19.2 

0.0 

Marital status (%) 

Married 

19.0 

18.6 

12.2 

0.7 

Unmarried 

70.1 

77.6 

81.5 

89.9 

Missing 

11.0 

3.8 

6.2 

9.4 

Number of children (%) 

0 

54.3 

0.0 

74.8 

98.2 

1 

16.7 

40.5 

9.7 

1.8 

2 

13.1 

30.1 

7.7 

0.0 

3 or more 

15.9 

29.5 

7.7 

0.0 

Race/ethnicity b (%) 

Flispanic 

30.3 

9.3 

99.8 

61.0 

White 

27.8 

55.3 

0.0 

22.2 

Black 

37.6 

31.8 

0.0 

2.2 

Asian 

0.5 

0.0 

0.0 

3.9 

Other 

3.8 

3.6 

0.2 

10.7 

Currently employed (%) 

51.5 

47.6 

43.9 

48.5 

First person in family to attend college (%) 

33.2 

28.7 

38.7 

33.5 

Financial aid in the first vear c 

Cost of attendance ($) 

15,834 

13,623 

9,985 

16,199 

Expected Family Contribution (EFC) 

832 

0 

647 

1,077 

Students with zero EFC (%) 

66.8 

100.0 

66.2 

48.8 

Gross need ($) 

14,818 

13,623 

9,349 

15,122 

Financial aid received ($) 

Average 

6,256 

9,342 

4,831 

10,330 

Maximum 

19,454 

18,406 

15,421 

27,722 

Unmet need ($) 

Average 

8,563 

4,281 

3,962 

4,792 

Maximum 

21,772 

20,639 

13,656 

20,210 

Sample size (total = 1,702) 

401 

339 

417 

545 
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Table 3 (continued) 


SOURCES: MDRC calculations using Baseline Information Form (BIF) data and financial aid data from Pima 
Community College, Hillsborough Community College, Owens Community College, and the University of New 
Mexico. 

NOTES: Missing values are only included in variable distributions for characteristics with more than 5 percent of 
the sample missing. 

Distributions may not sum to 100 percent because of rounding. 

a Female students are not eligible to participate in the program at Pima Community College; thus, gender was 
imputed and not explicitly asked on the BIF. 

b Respondents who said they were Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they were not Hispanic and chose more than one race are considered Multiracial. These 
respondents, combined with those who said they were American Indian or Alaskan Native or another 
race/ethnicity, are included in the category “Other.” 

Estimates of annual financial aid are based on full-year data when available. For some students, the financial 
aid year used begins after their first semester in the study. 
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Table 4 


Enrollment Outcomes of Control Group Students: 
Selected Performance-Based Scholarship Sites 


Characteristics 

Hillsborough 

Community 

College 

Owens 

Community 

College 

Pima 

Community 

College 

University 

of 

New Mexico 

Enrolled (%) 





First semester 

94.0 

95.9 

97.1 

99.1 

Second semester 

78.1 

82.3 

74.8 

91.2 

Third semester 

58.6 

62.8 


78.5 

Fourth semester 


51.6 


74.1 

Enrolled in 12 or more credits (%) 





First semester 

60.8 

72.6 

74.3 

98.3 

Second semester 

46.1 

58.7 

50.1 

88.4 

Third semester 

30.9 

38.3 


73.8 

Fourth semester 


30.7 


69.7 

Sample size (total = 1,702) 

401 

339 

417 

545 


SOURCES: MDRC calculations using transcript data from Hillsborough Community College, the Ohio 
Board of Regents, Pima Community College, and the University of New Mexico. 
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Table 5 


Linear Model of 

First-Year Financial Aid Values and Subsequent Academic Outcomes, 
Control Group Students Only: 

Selected Performance-Based Scholarship Sites 


Model 

Number 

of 

Students 

Estimated 
Change 
Per $1,000 

T-statistic 

R 2 

Total financial aid and persistence 

Hillsborough Community College 

401 

2.36 

3 *** 

0.04 

Owens Community College 

339 

3.48 

7 10 *** 

0.13 

Pima Community College 

417 

6.78 

12.05 *** 

0.26 

University of New Mexico 

545 

3.14 

7 34 *** 

0.09 

Total financial aid and credits attempted 

Hillsborough Community College 

401 

0.34 

4 76 

0.05 

Owens Community College 

339 

0.49 

6 95 *** 

0.13 

Pima Community College 

417 

1.24 

16.08 *** 

0.38 

University of New Mexico 

545 

0.61 

7 8 1 *** 

0.10 

Total financial aid and credits earned 

Hillsborough Community College 

401 

0.18 

2.86 *** 

0.02 

Owens Community College 

339 

0.35 

5.66 *** 

0.09 

Pima Community College 

417 

0.89 

1 1 ,44 *** 

0.24 

University of New Mexico 

545 

0.54 

7 08 *** 

0.08 

Unmet need and persistence 

Hillsborough Community College 

401 

-2.06 

-3 71 *** 

0.03 

Owens Community College 

339 

- 1.10 

- 2.22 ** 

0.01 

Pima Community College 

391 

-0.79 

-1.08 

0.00 

University of New Mexico 

545 

-3.90 

-8.80 *** 

0.12 

Unmet need and credits attempted 

Hillsborough Community College 

401 

-0.26 

-3 95 *** 

0.04 

Owens Community College 

339 

-0.09 

-1.23 

0.00 

Pima Community College 

391 

-0.14 

- 1.22 

0.00 

University of New Mexico 

545 

-0.78 

-9 71 *** 

0.15 

Unmet need and credits earned 

Hillsborough Community College 

401 

-0.18 

-3.01 *** 

0.02 

Owens Community College 

339 

-0.06 

- 1.01 

0.00 

Pima Community College 

391 

0.01 

0.10 

0.00 

University of New Mexico 

545 

-0.70 

-8.85 *** 

0.13 
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Table 5 (continued) 


SOURCES: MDRC calculations using transcript and financial aid data from Hillsborough Community College, 
Owens Community College, the Ohio Board of Regents, Pima Community College, and the University of New 
Mexico. 

NOTES: Linear regression models were run with first-year financial aid or unmet need as the dependent variable 
and persistence, credits attempted, or credits earned as the independent variable. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates of annual financial aid are based on full-year data when available. For some students, the financial aid 
year used begins after their first semester in the study. 

Academic outcomes for Owens Community College and the University of New Mexico are from students’ 
fourth semester in the study. For Hillsborough Community College, outcomes are from students’ third semester, 
and for Pima Community College outcomes are from students’ second semester. 

Unadjusted r 2 shown for all models. 
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Table 6 


Credits Earned in the First Year: 

Selected Performance-Based Scholarship Sites, by Subgroup 


Subgroup 

Sample 

Size 

Program 

Group 

Control 

Group Difference 

Standard 

Error 

Difference 

Between 

Subgroups 

Gender 

Male 

2,588 

17.5 

15.7 

| g *** 

0.4 

t 

Female 

4,383 

17.2 

16.2 

i o *** 

0.3 


Sample size 

6,971 






Hispanic 

Yes 

2,845 

17.9 

16.8 

1 1 *** 

0.4 


No 

4,033 

16.9 

15.4 

1 5 *** 

0.3 


Sample size 

6,878 






Parent 

Yes 

3,721 

15.5 

14.1 

1 4 *** 

0.3 


No 

3,212 

19.4 

18.3 

1 1 *** 

0.3 


Sample size 

6,933 






Younger than 20 years old 

Yes 

1,826 

21.6 

20.6 

1.0 ** 

0.5 


No 

5,145 

15.8 

14.4 

1 4 *** 

0.3 


Sample size 

6,971 






First in family to attend college 

Yes 

2,211 

16.9 

15.3 

1 ^ *** 

0.4 


No 

4,574 

17.5 

16.5 

| Q *** 

0.3 


Sample size 

6,785 






Employed at beginning of program 

Yes 

3,424 

17.2 

15.9 

2 2 *** 

0.3 


No 

3,413 

17.5 

16.2 

2 2 *** 

0.4 


Sample size 

6,837 






Program with services component 8 

Yes 

3,184 

19.0 

17.7 

1 2 *** 

0.4 


No 

3,787 

15.9 

14.6 

1 4 *** 

0.3 


Sample size 

6,971 
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Table 6 (continued) 


SOURCES: MDRC calculations using Baseline Information Form (BIF) data and transcript data from Pima 
Community College, Hillsborough Community College, the Ohio Board of Regents, the University of New 
Mexico, Borough of Manhattan Community College, and Hostos Community College. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between subgroups. Statistical significance levels are 
indicated as: ftt = 1 percent; ft = 5 percent; | = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by research cohort and campus. 

a The programs with services components are located in Arizona, Florida, and New Mexico. 


43 



Student income quartile 


Figure 1 

Average Unmet Need After Grant Aid Applied: 

Full-Time, Full-Year U.S Undergraduates by Student Income Quartile, 2003-2004 


Highest 


Upper middle 


Lower middle 


Lowest 



0 2,000 4,000 6,000 8,000 10,000 


Unmet need ($) 


■ Financially dependent students □ Financially independent students 


12,000 


SOURCE: Long and Riley (2007). 



Enrollmnent in the second semester (%) 


Figure 2 

Control Group Enrollment in the Second Semester by First-Year Unmet Need 

University of New Mexico 



First-year unmet need ($) 


SOURCES: MDRC calculations using transcript data and financial aid data from the University of New Mexico. 


Figure 3 

Total Financial Aid Received in the First Year: 
Selected Performance-Based Scholarship Sites 


12,000 

932 *** 


11,098 



Arizona New Mexico New York Ohio 

■ Program group □ C ontrol group 

SOURCES: MDRC calculations using financial aid data from Pima Community College, the University of New 
Mexico, the CUNY Institutional Research Database, Lorain County Community College, Owens Community 
College, and Sinclair Community College. 

NOTES: Students in the program groups were eligible for higher amounts of aid (as illustrated in Table 2) if they 
met all of the benchmarks of their programs. These amounts include only the performance-based scholarship 
awards that students actually earned. 

A two-tailed t-test was applied to differences between the research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by research cohort and campus. 

Financial aid outcomes for the New York study represent financial aid awarded, which is distinct from financial 
aid received. Financial aid outcomes for the Arizona, New Mexico, and Ohio studies represent financial aid 
received. 
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Figure 4 

Credits Earned in the First Year: 
Selected Performance-Based Scholarship Sites 
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0.9 : 
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15.5 


13.9 


13.9 


10 


7.7 


Opening 

Doors 

Louisiana 


Arizona 


Florida New Mexico New York Ohio 


■ Program group □ Control group 


SOURCES: MDRC calculations using transcript data from Delgado Community College, Louisiana 
Technical College, Pima Community College, Flillsborough Community College, the University of 
New Mexico, Borough of Manhattan Community College, Hostos Community College, and the Ohio 
Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by research cohort and campus. 

Only the first two cohorts are shown for Opening Doors Louisiana. 
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